Diagnosis support system, diagnosis support method, and storage medium

ABSTRACT

A diagnosis support system includes a processor. The processor is connected to a plurality of classifiers that are different in performance. The processor displays performance information of each of the classifiers side by side, receives a user&#39;s selection of the performance information displayed side by side, and inputs an input image to the classifier associated with the performance information selected by the user.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of International Patent ApplicationNo. PCT/JP2020/029636, having an international filing date of Aug. 3,2020, which designated the United States, the entirety of which isincorporated herein by reference.

BACKGROUND

In recent years, known is a diagnosis support method (computer aideddetection/diagnosis (CAD)) of indicating a position of a lesioncandidate and displaying differential information on moving imagescaptured using an endoscope. For example, International Publication No.2019/087791 discloses a method of performing learning on a newclassifier using an image held by a user, and, in a case where theclassifier improves in performance in comparison with a referenceclassifier, changing a classifier. Additionally, InternationalPublication No. 2018/180573 discloses a system of comparing imagesbefore and after updating of image processing software.

SUMMARY

In accordance with one of some aspect, there is provided a diagnosissupport system comprising a processor, the processor being connected toa plurality of classifiers that are different in performance; theprocessor displaying performance information of each of the plurality ofclassifiers side by side; receiving a user's selection of theperformance information displayed side by side; and inputting an inputimage to a corresponding one of the plurality of classifiers, thecorresponding one being associated with the performance informationselected by the user.

In accordance with one of some aspect, there is provided a diagnosissupport method comprising: presenting performance information that isinformation regarding performance of a plurality of classifiers, theplurality of classifiers outputting mutually different detection resultswhen detecting a region of interest from an input image; receiving auser's selection for selecting at least one of the plurality ofclassifiers as a classifier serving as an output target; and outputtinga detection result of the classifier selected by the user's selection,the presenting including presenting at least two types of performance ina trade-off relationship as the performance information.

In accordance with one of some aspect, there is provided a storagemedium storing a diagnosis support program that causes a computer toimplement: presenting performance information that is informationregarding performance of a plurality of classifiers, the plurality ofclassifiers outputting mutually different detection results whendetecting a region of interest from an input image; receiving a user'sselection for selecting at least one of the plurality of classifiers asa classifier serving as an output target; and outputting a detectionresult of the classifier selected by the user's selection.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a configuration example of a diagnosis supportsystem.

FIG. 2 illustrates a configuration example of an endoscope systemincluding the diagnosis support system.

FIG. 3 is a flowchart describing processing in the diagnosis supportsystem.

FIGS. 4A and 4B are diagrams for describing a neural network.

FIG. 5 is a flowchart describing a learning process.

FIG. 6 illustrates an example of a method of acquiring an originalclassifier and a customized classifier.

FIGS. 7A to 7C each illustrate an example of data used for generationand evaluation of the customized classifier.

FIG. 8 is a diagram for describing processing performed in a classifiercandidate.

FIG. 9 illustrates an example of a screen displaying a detection resultbased on an input image.

FIG. 10 illustrates an example of a screen displaying a detection resultbased on an input image.

FIG. 11 illustrates an example of a screen displaying a list ofdetectable lesions.

FIGS. 12A to 12C each illustrate an example of a screen displaying alist of classification performance such as a recall rate.

FIG. 13 illustrates an example of a screen displaying classificationperformance such as a recall rate.

FIG. 14 illustrates an example of a screen displaying learning data.

FIG. 15 is another flowchart describing processing in the diagnosissupport system.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

The following disclosure provides many different embodiments, orexamples, for implementing different features of the provided subjectmatter. These are, of course, merely examples and are not intended to belimiting. In addition, the disclosure may repeat reference numeralsand/or letters in the various examples. This repetition is for thepurpose of simplicity and clarity and does not in itself dictate arelationship between the various embodiments and/or configurationsdiscussed. Further, when a first element is described as being“connected” or “coupled” to a second element, such description includesembodiments in which the first and second elements are directlyconnected or coupled to each other, and also includes embodiments inwhich the first and second elements are indirectly connected or coupledto each other with one or more other intervening elements in between.

Exemplary embodiments are described below. Note that the followingexemplary embodiments do not in any way limit the scope of the contentdefined by the claims laid out herein. Note also that all of theelements described in the present embodiment should not necessarily betaken as essential elements.

1. System Configuration Example

In computer aided detection/diagnosis (CAD), conventionally known are amethod of changing a detection sensitivity to a lesion and a method ofchanging processing depending on an organ. For example, in a case whereit is determined that the lesion has been detected on condition that adegree of reliability as an output of a trained model is larger than orequal to a given threshold, it is possible to change a detectionsensitivity by changing the threshold. For example, in a case whereexperienced doctors are targeted, a reduction in detection sensitivitymakes a lesion harder to be detected, whereby it is possible to preventissuance of unnecessary notification. In a case where novice doctors aretargeted, an increase in detection sensitivity makes a lesion easier tobe detected, whereby it is possible to prevent a doctor from overlookingthe lesion. However, what kind of lesion about which notification isdesired to be made largely depends on a user's preference. In a casewhere the threshold has been changed, it is difficult to preliminarilygrasp how a detection result changes. Hence, it is not easy to make asensitivity setting that reflects the user's preference.

In a case where a trained model is generated for each organ, generatedare a trained model that is suited to detection of a lesion in thestomach and a trained model that is suited to detection of a lesion inthe intestine. By switching the trained model to be used depending on anorgan serving as an observation target, detection accuracy is expectedto increase. However, merely switching the trained model depending on apart cannot sufficiently reflect the user's preference. This is because,even in a case where an identical part is observed, what kind of lesionabout which notification is desired to be made is different depending ona user.

The method in accordance with International Publication No. 2019/087791is to compare performance of a plurality of classifiers to determinewhether or not to update a classifier. However, the user's preference isnot reflected on determination about whether or not the performance isimproved. In addition, the user cannot grasp how the detection resultspecifically changes before and after the updating. The method inaccordance with International Publication No. 2018/180573 is to displayimages before and after updating of image processing software.International Publication No. 2018/180573 relates to a technique ofsimultaneously displaying a plurality of images as processing results,and never discloses a classifier that detects a region of interest froman image, let alone a method of performing display regarding performanceof the classifier.

FIG. 1 is a diagram illustrating a configuration of a diagnosis supportsystem 100 in accordance with the present embodiment. The diagnosissupport system 100 includes a classification section 110, a performanceinformation processing section 120, and a user's selection receivingsection 130. However, a configuration of the diagnosis support system100 is not limited to that illustrated in FIG. 1 . Part of theconfiguration may be omitted, and various modification can be made, suchas addition of another configuration. For example, the diagnosis supportsystem 100 may include a configuration of a processing device 330, whichwill be described later with reference to FIG. 2 , or may include alearning section 210, which will be described later with reference toFIG. 6 .

The classification section 110 is capable of outputting a plurality ofdetection results based on a plurality of classifier candidates thatoutputs mutually different detection results when detecting the regionof interest from an input image. The input image in the presentembodiment is, specifically, an in-vivo image in which the living bodyis captured. Note that the region of interest in the present embodimentis a region in which the order of priority in observation for the useris relatively higher than that in other regions. In a case where theuser is a doctor who makes a diagnosis or performs a treatment, theregion of interest corresponds to, for example, a region that shows alesion portion. Note that if a target the doctor wants to observe isbubbles or residues, the region of interest may be a region that shows aportion of the bubbles or a portion of the residues. That is, while atarget to which the user should pay attention is different depending ona purpose of observation, a region where the order of priority inobservation for the user is relatively higher than that in the otherregions is the region of interest. The following description will begiven of an example in which the region of interest is a regioncorresponding to a lesion.

A plurality of classifier candidates mentioned herein is a plurality ofmutually different trained models that can be acquired by the diagnosissupport system 100. The plurality of trained models may be stored in astorage section of the diagnosis support system 100 or may be acquiredfrom an external device with use of a communication section. The storagesection and the communication section are not illustrated. Theclassification section 110 is capable of switching the detection resultas an output by switching which of the plurality of trained models theclassification section 110 follows to operate. As described later withreference to FIGS. 9 and 10 , the classification section 110 is notprevented from outputting a plurality of detection resultssimultaneously. As described later with reference to FIG. 8 , theclassifier candidate is not limited to the trained model alone, and maybe a combination of at least one of pre-processing or post-processingand the trained model.

The performance information processing section 120 performs a process ofdisplaying performance information serving as information regardingperformance of a plurality of classifier candidates. For example, theperformance information processing section 120 is a processor thatperforms display control, and performs a process of generating a displayimage and control of causing a display section to display the displayimage. Details of a screen that displays the performance informationwill be described later with reference to FIGS. 9 to 14 . Note that theperformance information processing section 120 may perform a process ofgenerating the performance information or perform a process of acquiringthe performance information generated in an external device.

The user's selection receiving section 130 receives a selectionoperation performed by the user as a user's selection. Specifically, theuser's selection receiving section 130 receives the user's selection forselecting at least one of the plurality of classifier candidates as aclassifier serving as an output target. The user's selection receivingsection 130 is, for example, a processor that controls an operationinterface, which is not illustrated. As the operation interface, variouskinds of interfaces, such as a mouse, a keyboard, a touch panel, abutton, a lever, and a knob, can be used.

The classification section 110 in accordance with the present embodimentoutputs a detection result of a classifier selected by the user'sselection. In accordance with the method of the present embodiment, thediagnosis support system 100 presents performance information of aplurality of classifiers to the user and then outputs a detection resultof the region of interest using the classifier selected by the user.Since the user sees the performance information and then selects theclassifier, he/she is able to select the classifier that is suited tohis/her preference. Hence, the diagnosis support system 100 is capableof outputting the detection result that is suited to the user'spreference.

Note that the diagnosis support system 100 in accordance with thepresent embodiment is composed of the following hardware. The hardwarecan include at least one of a circuit that processes a digital signal ora circuit that processes an analog signal. For example, the hardware caninclude one or more circuit devices mounted on a circuit board, or oneor more circuit elements. The one or more circuit devices are, forexample, integrated circuits (ICs), field-programmable gate array (FPGA)circuits, or the like. The one or more circuit elements are, forexample, resistors, capacitors, or the like.

Each section of the diagnosis support system 100 may be implemented bythe following processor. The diagnosis support system 100 includes amemory that stores information, and a processor that operates based onthe information stored in the memory. The information is, for example, aprogram, various kinds of data, and the like. The processor includeshardware. Note that various kinds of processors such as a centralprocessing unit (CPU), a graphics processing unit (GPU), and a digitalsignal processor (DSP) can be used. The memory may be a semiconductormemory such as a static random access memory (SRAM) and a dynamic randomaccess memory (DRAM). The memory may be a register. The memory may be amagnetic storage device such as a hard disk drive (HDD). The memory maybe an optical storage device such as an optical disk device. Forexample, the memory stores a computer-readable instruction. Theinstruction is executed by the processor, whereby functions of sectionsof the diagnosis support system 100 is implemented as processing. Thediagnosis support system 100 includes, for example, the classificationsection 110, the performance information processing section 120, and theuser's selection receiving section 130, which are illustrated in FIG. 1. The instruction mentioned herein may be an instruction of aninstruction set that is included in a program, or may be an instructionthat instructs a hardware circuit included in the processor to operate.Furthermore, all or part of the sections of the diagnosis support system100 can be implemented by cloud computing, and each processing, whichwill be described later, can be executed on the cloud computing.

The diagnosis support system 100 in accordance with the presentembodiment, for example, may be included in an endoscope system 300.FIG. 2 is a diagram illustrating a configuration example of theendoscope system 300 including the diagnosis support system 100.

The endoscope system 300 includes an insertion section 310, a processingdevice 330, a display section 340, and a light source device 350.However, the configuration of the endoscope system 300 is not limited tothat illustrated in FIG. 2 . The configuration can be modified invarious manners, such as omission of part of the configuration andaddition of another configuration.

The light source device 350 includes a light source 352 that emitsillumination light. The light source 352 may be a xenon light source, alight emitting diode (LED), or a laser light source. Alternatively, thelight source 352 may be another light source, and a light emissionmethod is not limited.

The insertion section 310 includes an objective optical system 311, animage sensor 312, an illumination lens 314, and a light guide 315. Thelight guide 315 guides illumination light emitted from the light source352 to a leading end of the insertion section 310. The illumination lens314 emits illumination light guided by the light guide 315 onto anobject. The objective optical system 311 receives reflected light fromthe object and forms an image as an object image. The objective opticalsystem 311 includes, for example, a focus lens, and may be capable ofchanging a position at which the object image is formed in accordancewith a position of the focus lens. For example, the insertion section310 includes an actuator that drives the focus lens based on control bya control section 332. The actuator is not illustrated. The controlsection 332 performs autofocus (AF) control.

The image sensor 312 receives light from the object having passedthrough the objective optical system 311. The image sensor 312 may be amonochrome sensor, or may be an element having a color filter. The colorfilter may be a filter in a well-known Bayer's arrangement, acomplementary color filter, or another filter. The complementary colorfilter includes filters in respective colors of cyan, magenta, andyellow.

The processing device 330 performs image processing and control of thewhole system. The diagnosis support system 100 in accordance with thepresent embodiment is, for example, included in the processing device330. The processing device 330 includes the classification section 110,the performance information processing section 120, the user's selectionreceiving section 130, an image acquisition section 331, the controlsection 332, a storage section 333, and a display processing section336.

The processing device 330 is, for example, one device that is connectedto the insertion section 310 via a connector, but is not limitedthereto. For example, a configuration of part or the whole of theprocessing device 330 may be structured by another informationprocessing device such as a personal computer (PC) and a server systemthat can be connected via a network. For example, the processing device330 may be implemented by cloud computing. The network mentioned hereinmay be a private network such as an intranet, or may be a publictelecommunication network such as the Internet. In addition, the networkmay be a wired network or a wireless network. That is, the diagnosissupport system 100 may also be implemented as one device or implementedby distributed processing by a plurality of devices.

The image acquisition section 331 acquires image data captured by theimage sensor 312. The image acquisition section 331 performsanalog/digital (A/D) conversion for converting analog signals, which aresequentially output from the image sensor 312, to digital images, andperforms a correction process of various kinds on image data after theA/D conversion. Note that the image sensor 312 is provided with an A/Dconversion circuit, and the A/D conversion in the image acquisitionsection 331 may be omitted. Examples of the correction process mentionedherein include a color matrix correction process, a structureenhancement process, a noise reduction process, and automatic gaincontrol (AGC). The image acquisition section 331 may perform anothercorrection process such as a white balance process. The imageacquisition section 331 outputs the processed image to theclassification section 110 as an input image. In addition, the imageacquisition section 331 outputs the processed image to the displayprocessing section 336.

The performance information processing section 120 performs a process ofdisplaying the performance information. Specifically, the performanceinformation processing section 120 performs a process of generating adisplay screen for displaying the performance information and causingthe display section 340 to display the display screen.

The user's selection receiving section 130 receives operationinformation representing an operation input to an operating section. Theoperating section mentioned herein is for the user to perform variousoperations on the endoscope system 300, and is implemented by variouskinds of buttons, a graphical user interface (GUI), or the like. Theoperating section may include, for example, a knob for operating bendingof the leading end of the insertion section 310, a button forcontrolling the start/end of AF, and the like. The user's selectionreceiving section 130 receives the user's selection operation withrespect to display of the performance information.

The storage section 333 is a work area for the control section 332, theclassification section 110, and the like, and a function thereof can beimplemented by a memory such as a static random access memory (SRAM) anda dynamic random access memory (DRAM), an HDD, or the like. The storagesection 333 stores, for example, a plurality of trained models that isdifferent in output.

The classification section 110 performs a process for detecting theregion of interest from the input image. Specifically, theclassification section 110 performs a process of identifying a trainedmodel based on the user's selection that is made regarding display ofthe performance information and that is received by the user's selectionreceiving section 130. The classification section 110 operates inaccordance with the identified trained model to perform the process ofdetecting the region of interest from the input image, and thereafteroutputs the detection result to the display processing section 336. Inaddition, the classification section 110 may output a degree ofreliability representing a degree of probability of the detected regionof interest.

The display processing section 336 performs processing based on theimage from the image acquisition section 331 and the detection resultfrom the classification section 110, and performs a process ofoutputting a processing result to the display section 340. For example,the display processing section 336 may perform a process of adding thedetection result from the classification section 110 to the image fromthe image acquisition section 331, and displaying the image to which thedetection result is added.

The control section 332 is connected to each of the classificationsection 110, the performance information processing section 120, theuser's selection receiving section 130, the image sensor 312, the imageacquisition section 331, the display processing section 336, and thelight source 352, and controls each section.

The display section 340 is, for example, a liquid crystal display, anelectro-luminescence (EL) display, or the like.

FIG. 3 is a flowchart describing the outline of processing of thediagnosis support system 100 in accordance with the present embodiment.This processing is started, for example, by power-on of the diagnosissupport system 100. For example, when each section of the endoscopesystem 300 including the diagnosis support system 100 transitions to anoperable state, the following processing is executed.

First, in step S101, whether or not an operation mode of the diagnosissupport system 100 is a classifier selection mode is determined.Processing in step S101 may be performed by the classification section110 or may be performed by the control section 332. The classifierselection mode is a mode for displaying the performance information toreceive the user's selection of the classifier used for outputting ofthe detection result.

In a case of YES in step S101, in step S102, the performance informationprocessing section 120 performs a process of displaying the performanceinformation of classifier candidates. In step S103, the user's selectionreceiving section 130 performs a process of receiving the user'sselection for selecting any of the classifier candidates. Theclassification section 110 identifies the classifier candidate selectedby the user as the classifier used for outputting of the detectionresult.

In a case of NO in step S101, the user's selection receiving section 130does not receive the user's selection for selecting a classifier. Inthis case, for example, the classification section 110 identifies adefault classifier candidate as the classifier used for outputting ofthe detection result.

After the processing in step S103 or step S104, in step S105, thediagnosis support system 100 starts an operation in an observation mode.The observation mode is a mode for inserting the insertion section 310into the inside of a living body to capture an in-vivo image. Theobservation mode can be referred in other words to as a mode in whichthe user such as a doctor observes the inside of the living body of apatient based on the in-vivo image. In the observation mode, the imageacquisition section 331 sequentially acquires time-series in-vivo imagescaptured by the image sensor 312, and outputs the in-vivo images to theclassification section 110. The classification section 110 inputs thein-vivo images to the classifier identified in step S104 or step S105,and acquires and outputs the detection result of the region of interest.

The method in accordance with the present embodiment may be implementedas a diagnosis support method. The diagnosis support method includesacquiring the performance information serving as information regardingperformance of the plurality of classifier candidates that outputsmutually different detection results when detecting the region ofinterest from the input image, presenting the acquired performanceinformation to the user, receiving the user's selection for selecting atleast one of the plurality of classifier candidates as the classifierserving as the output target, and outputting the detection result of theclassifier candidate selected by the user's selection.

The method in accordance with the present embodiment may be applied to aprogram that implements processing performed by the diagnosis supportsystem 100. The program can be stored, for example, in an informationstorage device, which is a computer readable storage medium. Theinformation storage medium is implemented by, for example, an opticaldisk, a memory card, an HDD, or a semiconductor memory. Thesemiconductor memory is, for example, a read-only memory (ROM). Thediagnosis support system 100 performs various kinds of processing inaccordance with the present embodiment based on the program stored inthe information storage device. That is, the information storage devicestores the program for causing the computer to function as each sectionof the diagnosis support system 100. The computer is a device providedwith an input device, a processing section, a storage section, and anoutput section. Specifically, the program in accordance with the presentembodiment is a diagnosis support program for causing the computer toexecute each step, which will be described later with reference to FIG.3 .

The diagnosis support program causes the computer to acquire theperformance information serving as information regarding performance ofthe plurality of classifier candidates that outputs mutually differentdetection results when detecting the region of interest from the inputimage, present the acquired performance information to the user, receivethe user's selection for selecting at least one of the plurality ofclassifier candidates as the classifier serving as the output target,and output the detection result of the classifier candidate selected bythe user's selection. For example, each section of the diagnosis supportsystem 100 in accordance with the present embodiment is implemented as amodule of a program that operates on a processor. The processor includeshardware. The classification section 110 is implemented as an imageprocessing module for detecting the region of interest from the inputimage based on the classifier. The performance information processingsection 120 is implemented as a display control module for displayingthe performance information. The user's selection receiving section 130is implemented as an interface control module for receiving operationinformation indicating the user's selection.

2. Example of a Plurality of Classifier Candidates

As described above, the diagnosis support system 100 in accordance withthe present embodiment is capable of outputting a plurality of detectionresults based on the plurality of classifier candidates. An example ofthe plurality of classifiers that outputs mutually different detectionresults will be described in detail below. Note that the followingdescription will be given of an example in which the classifier includesa trained model acquired by machine learning. However, the classifiermay be an image processing algorithm or the like generated without usingthe machine learning.

2. 1 Example of Trained Model

The classifier in accordance with the present embodiment is, forexample, a trained model that performs a process of detecting a lesionfrom an input image, and that outputs a detection result. The machinelearning in accordance with the present embodiment is, for example,supervised learning. One piece of learning data used for the machinelearning is data in which a piece of input data and a correct labelcorresponding to the piece of input data are associated with each other.The input data is a learning image. The correct label is informationthat identifies the lesion in the learning image. The correct label maybe information that identifies the presence/absence of the lesion, aposition of the lesion, and a size of the lesion. The classifier inaccordance with the present embodiment may classify the lesion. Thecorrect label in this case includes information that identifies a resultof classifying the lesion. The result of classification is, for example,a result of classification in accordance with a degree of malignancy ofthe lesion. The correct label is, for example, a result of annotationadded by a user who has expert knowledge, such as a doctor.

The outline of the machine learning is now described. The followingdescription is given of the machine learning using a neural network, butthe method in accordance with the present embodiment is not limitedthereto. In the present embodiment, for example, machine learning usinganother model such as a support vector machine (SVM) may be performed,and machine learning using a method that has developed from variousmethods such as the neural network and the SVM may be performed.

FIG. 4A is a schematic diagram for describing the neural network. Theneural network includes an input layer that takes input data, anintermediate layer that executes calculation based on an output from theinput layer, and an output layer that outputs data based on an outputfrom the intermediate layer. While FIG. 4A exemplifies a network havingthe intermediate layer composed of two layers, the intermediate layermay be composed of one layer, or three or more layers. In addition, thenumber of nodes included in each layer is not limited to that in theexample of FIG. 4A, and can be modified in various manners. Note that inconsideration of accuracy, learning in accordance with the presentembodiment is preferably performed using deep learning using amulti-layer neural network. The multi-layer mentioned herein means fouror more layers in a more limited sense.

As illustrated in FIG. 4A, a node included in a given layer is connectedto a node in an adjacent layer. A weight coefficient is assigned betweenconnected nodes. Each node multiplies an output from a node in a formerstage by the weight coefficient and obtains a total value of results ofmultiplication. Furthermore, each node adds a bias to the total valueand applies an activation function to a result of addition to obtain anoutput from the node. This processing is sequentially executed from theinput layer to the output layer, whereby an output from the neuralnetwork is obtained. Note that as the activation function, various kindsof functions such as a sigmoid function and a rectified linear unit(ReLU) function are known, and a wide range of these functions can beapplied in the present embodiment.

The learning in the neural network is a process of determining anappropriate weight coefficient. The weight coefficient mentioned hereinincludes a bias. The following description is given of an example inwhich a process of generating the trained model is performed in alearning device. The learning device may be a learning section 210 or alearning device 220, which will be described later with reference toFIG. 6 .

The training device inputs input data out of learning data to the neuralnetwork and performs calculation in a forward direction using a weightcoefficient at this time to obtain an output. The learning devicecalculates an error function based on the output and the correct labelout of the learning data. The learning device updates the weightcoefficient to make the error function smaller. In updating the weightcoefficient, for example, backpropagation to update the weightcoefficient from the output layer to the input layer can be utilized.

The neural network may be, for example, a convolutional neural network(CNN). FIG. 4B is a schematic diagram for describing the CNN. The CNNincludes a convolution layer that performs convolution calculation and apooling layer. The convolution layer is a layer that performs a filterprocess. The pooling layer is a layer that reduces a size in a verticaldirection and a size in a lateral direction to perform poolingcalculation. In the example illustrated in FIG. 4B, the CNN is a networkthat causes each of the convolution layer and the pooling layer toperform calculation multiple times, thereafter causes a fully connectedlayer to perform calculation, and thereby obtain an output. The fullyconnected layer is a layer that performs a calculation process in a casewhere all nodes included in the former layer are connected tocorresponding nodes in the given layer, and the calculation processcorresponds to calculation in each layer described above with referenceto FIG. 4A. Note that also in a case where the CNN is used, although notillustrated in FIG. 4B, the calculation process with the activationfunction is performed similarly to the case in FIG. 4A. Variousconfigurations of the CNN have been known, and a wide range of theseconfigurations are applicable to the present embodiment.

Also in the case where the CNN is used, a procedure of processing issimilar to that illustrated in FIG. 4A. That is, the learning deviceinputs input data, out of the learning data, to the CNN, and performs afilter process or pooling calculation using filter characteristics atthat time to obtain an output. The learning device calculates the errorfunction based on the output and the correct label, and updates theweight coefficient including the filter characteristics to make theerror function smaller. For example, the backpropagation can be utilizedalso when the weight coefficient of the CNN is updated.

FIG. 5 is a schematic diagram for describing learning processing in theneural network. First, in steps S201 and S202, the learning deviceacquires the learning image and the correct label associated with thelearning image. For example, the learning device acquires multitudes ofpieces of data in which the learning image and the correct label areassociated with each other, and stores the data as learning data. Theprocessing in each of steps S201 and S202 is, for example, a process ofreading out one piece of data out of the learning data.

In step S203, the learning device performs a process of calculating anerror function. Specifically, the learning device inputs the learningimage to the neural network and performs calculation in the forwarddirection based on a weight coefficient at this time. The learningdevice then calculates the error function based on a process ofcomparing a calculation result and the correct label with each other.Furthermore, in step S203, the learning device performs a process ofupdating the weight coefficient to make the error function smaller. Thebackpropagation or the like can be utilized for this process, asdescribed above. The processing in steps S201 to S203 corresponds to aone-time learning process based on one piece of learning data.

In step S204, the learning device determines whether to end the learningprocess. For example, the learning device may retain part of multitudesof learning data as evaluation data. The evaluation data is data forchecking accuracy of a learning result, and data that is not used forupdating the weight coefficient. In a case where a rate of correctanswers in an estimation process using the evaluation data exceeds apredetermined threshold, the learning device ends the learning process.

In a case of NO in step S204, the processing returns to step S201, andthe learning process based on subsequent learning data continues. In acase of YES in step S204, the learning process ends. The learning devicetransmits information of the generated trained model to the diagnosissupport system 100. In the example illustrated in FIG. 2 , theinformation of the trained model is stored in the storage section 333.Various kinds of methods for the machine learning such as batch learningand mini-batch learning have been known, and a wide range of thesemethods are applicable to the present embodiment.

2.2 Original Classifier Candidate and Customized Classifier Candidate

A plurality of classifier candidates that outputs different detectionresults may include an original classifier candidate and a customizedclassifier candidate. The original classifier candidate is, for example,a classifier candidate that comes with the diagnosis support system 100when the diagnosis support system 100 is provided. For example, theoriginal classifier candidate is a classifier candidate that isgenerated by a manufacturer that provides the diagnosis support system100 or the like. For example, in a case where the diagnosis supportsystem 100 is utilized in a plurality of hospitals, the originalclassifier candidate is provided in common to the plurality ofhospitals.

Meanwhile, the customized classifier candidate is a classifier candidategenerated by a user based on an image acquired by the user. For example,in each hospital, iv-vivo images are acquired and accumulated using theendoscope system 300. The customized classifier candidate is generatedby machine learning using each in-vivo image as the learning image.Hence, the customized classifier candidate is a classifier candidatethat is different depending on a hospital.

FIG. 6 is a diagram for describing a configuration example of a systemusing the original classifier candidate and the customized classifiercandidate. The system includes the insertion section 310, the processingdevice 330, the learning device 220, and an endoscope system for imagecollection 400. The processing device 330 includes the diagnosis supportsystem 100 and the learning section 210. Note that a configuration ofthe system is not limited to the configuration illustrated in FIG. 6 .The configuration can be modified in various manners such as partialomission of constituent elements and addition of another constituentelement. The processing device 330 is the processing device 330illustrated in FIG. 2 , and may be implemented as one device orimplemented by distributed processing by a plurality of devices. Theinsertion section 310 and the processing device 330 correspond to theendoscope system 300 illustrated in FIG. 2 .

The endoscope system for image collection 400 captures a plurality ofin-vivo images for creating the original classifier candidate. Thelearning device 220 acquires a pair of the learning image captured bythe endoscope system for image collection 400 and a result of annotationadded to the learning image as the learning data used for the machinelearning. The learning data includes original learning data and originalevaluation data. The learning device 220 performs the machine learningbased on the original learning data to generate a trained modelcorresponding to the original classifier candidate. The learning device220 evaluates the generated trained model based on the originalevaluation data. The trained model is, for example, a model thatperforms an inference process in accordance with deep learning, asdescribed above. The learning device 220 transmits the generated trainedmodel to the diagnosis support system 100 via a network NW. The networkNW mentioned herein may be a public telecommunication network such asthe Internet or a private network.

Meanwhile, the endoscope system 300 captures a plurality of in-vivoimages using the insertion section 310. The endoscope system 300acquires a correct label corresponding to each of the plurality ofin-vivo images, and stores data in which the in-vivo image and thecorrect label are associated with each other as customized learning datain the storage section 333. The learning section 210 performs theprocessing illustrated in FIG. 5 based on the customized learning datato generate a trained model corresponding to the customized classifiercandidate. The learning section 210 transmits the generated trainedmodel to the diagnosis support system 100.

This allows the diagnosis support system 100 to execute a detectionprocess based on the original classifier candidate and the customizedclassifier candidate that are different in learning data.

FIGS. 7A to 7C are diagrams each illustrating a method of generating thecustomized classifier candidate based on the customized image data. Asillustrated in FIG. 7A, the customized classifier candidate may begenerated based on both the original learning data and the customizedlearning data. Then, a customized classifier is evaluated based on theoriginal evaluation data. Alternatively, as illustrated in FIG. 7B, thecustomized classifier candidate may be generated based on the customizedlearning data. In an example illustrated in FIG. 7B, the customizedclassifier candidate is evaluated using the original evaluation datasimilarly to an example illustrated in FIG. 7A.

Alternatively, as illustrated in FIG. 7C, the customized classifiercandidate may be generated and evaluated based on the customizedlearning data. For example, the customized learning data is divided intofour pieces of customized learning data. The learning section 210repeats a process of generating the customized classifier candidatebased on three pieces of customized learning data out of the four piecesof customized learning data, and evaluating the generated customizedclassifier candidate based on the remaining one piece of customizedlearning data, four times. Execution of cross-validation enablesappropriate generation and evaluation of the customized classifiercandidate based on the customized learning data.

As described above, the machine learning is only required to beperformed on the customized classifier candidate using the customizedlearning data that is not used for learning of the original classifiercandidate, and whether or not the original learning data is used and howto evaluate the customized classifier candidate can be modified invarious manners. In any methods, with use of the customized learningdata, a detection result of the customized classifier candidate isexpected to be different from a detection result of the originalclassifier candidate.

As described above, the plurality of classifier candidates in accordancewith the present exemplary embodiment may include the originalclassifier candidate and the customized classifier candidate created bythe user. This allows the user to add his/her original classifiercandidate.

As described above, the method in accordance with the present embodimentis to output a detection result for the region of interest that reflectsthe user's preference. However, it is not realistic to create anexhaustive original classifier candidate that satisfies a preference ofevery user. In this regard, allowing the user to add a classifiercandidate can increase a probability for the presence of a classifiercandidate that is suited to the user's preference.

The customized classifier candidate is created based on the machinelearning using learning images including an image held by the user, asillustrated in FIGS. 7A to 7C. The image held by the user is, forexample, an in-vivo image that has been previously set by the user as atarget of observation or treatment. Hence, it is presumed that the userhas many opportunities for performing observation targeting at a regionof interest of a similar kind. Use of the image held by the user inlearning of the customized classifier candidate enables creation of theclassifier candidate having a higher probability for being suited to theuser's preference.

2. 3 Difference Between Models

The plurality of classifier candidates is not limited to those that aredifferent in learning data used for learning as described above. Forexample, the plurality of classifier candidates may be classifiercandidates whose models are mutually different. The model mentionedherein represents, for example, the configuration of the neural networkillustrated in FIGS. 4A and 4B. The models being different specificallymeans that at least one of pieces of information that determinescharacteristics of the neural network is different. Examples of theinformation that determines characteristics include the number ofintermediate layers, the number of nodes included in each layer, aconnection relationship between a node in a given layer and a node in anext layer, and an activation function. For example, as a CNN thatdetects a given object, various kinds of models such as a FasterRegion-based CNN (Faster R-CNN), You only Look Once (YOLO), and a SingleShot Detector (SSD), have been known. The two models being differentmeans, for example, that one of the two models is the YOLO, and theother thereof is the SSD.

As described above, in a case where the learning data is different amonga plurality of trained models, different trained models are generatedeven from an identical model. Specifically, a calculation algorithm inthe forward direction is identical but weight coefficient information isdifferent from each other, whereby the plurality of trained models thatoutputs different detection results with respect to an identical inputimage is generated.

In a case where models are different from each other, different trainedmodels are generated even if the learning data is identical.Specifically, since calculation algorithms in the forward direction aredifferent from each other in the first place, the plurality of trainedmodels that outputs different detection results from the identical inputimage is generated.

As a matter of course, the plurality of classifier candidates may bedifferent in both learning data and model. For example, the originalclassifier candidate and the customized classifier candidate describedabove may be different not only in learning data used for learning butalso in the model itself.

Note that “the detection result being different” in the presentembodiment means that at least one of the learning data or the model isonly required to be different, as described above. In other words, “thedetection result being different” means that the difference incalculation algorithm in the forward direction or weight coefficientinformation differentiates processing to be executed in the input image.Hence, depending on a specific trained model or an input image,incidental coincidence of detection results from the plurality ofclassifier candidates is not excluded.

2. 4 Pre-Processing and Post-Processing

As described above, the classifier candidate is the trained model in alimited sense. That is, the storage section stores information regardingthe trained model. The classification section 110 serving as a processoroperates in accordance with an instruction from the trained model readout from the storage section to perform the process of detecting theregion of interest from the input image.

Calculation in accordance with the trained model in the classificationsection 110, that is, calculation for outputting output data based oninput data, may be executed by software, or may be executed by hardware.In other words, product-sum calculation executed in each node in FIG.4A, or a filter process or the like executed in the convolution layer ofthe CNN may be executed by software. Alternatively, the above-mentionedcalculation may be executed by a circuit device such as the FPGAcircuit. Still alternatively, the above-mentioned calculation may beexecuted by software and hardware in combination. In this manner,operations of the classification section 110 in accordance with aninstruction from the trained model can be implemented in variousmanners. For example, the trained model includes an inference algorithm,and a weight coefficient used in the inference algorithm. The inferencealgorithm is an algorithm for performing filter calculation or the likebased on input data. In this case, both the inference algorithm and theweight coefficient are stored in the storage section, and theclassification section 110 may read out the inference algorithm and theweight coefficient to perform an inference process with software. Thestorage section is, for example, the storage section 333 of theprocessing device 330, but another storage section may be used.Alternatively, the inference algorithm may be implemented by the FPGAcircuit or the like, and the storage section may store the weightcoefficient. Still alternatively, the inference algorithm including theweight coefficient may be implemented by the FPGA circuit or the like.In this case, the storage section that stores information of the trainedmodel is, for example, a built-in memory of the FPGA circuit.

The classifier candidate in accordance with the present embodiment is,for example, the trained model itself. For example, the detection resultof the classifier candidate corresponds to an output when the inputimage is input to the trained model. However, the classifier candidateis not limited to the trained model alone.

FIG. 8 is a diagram for processing in the classifier candidate. Asillustrated in FIG. 8 , in the classifier candidate, some kind ofpre-processing is performed on the input image, and a result of thepre-processing is input to the trained model. In addition, a result ofsome kind of post-processing performed on the output of the trainedmodel is output as the detection result. Note that either thepre-processing or the post-processing in FIG. 8 may be omitted.

For example, a flat lesion has few characteristics such as edges, and islikely to be buried in noise. In this case, execution of pre-processingfor integrating input images in a plurality of frames reduces noise, andcan thereby increase accuracy of detecting a lesion. However, sincethere is a need for integrating the plurality of frames, there occurs atime lag until the detection result is output, and a processing loadincreases. In this manner, there is a trade-off relationship betweenaccuracy and time when the lesion is detected, and which of settings isbetter depends on the user's preference.

Hence, the plurality of classifier candidates in accordance with thepresent embodiment may include a plurality of classifiers whose trainedmodels are identical and that have been subjected to different kinds ofpre-processing. In the above-mentioned example, a first classifiercandidate inputs input images to the trained model without execution offrame integration. A second classifier candidate executes the frameintegration of input images in a predetermined number of frames as thepre-processing, and inputs images after the pre-processing to thetrained model. This enables generation of the plurality of classifiercandidates based on one trained model.

Additionally, on the condition that a lesion that is determined to beidentical is consecutively detected in the predetermined number or moreframes, the endoscope system 300 outputs a detection result regardingthe lesion. This is because, since such a lesion as that appears onlyfor a short period of time and soon disappears from a screen is excludedfrom a display target, it is possible to prevent frequent change of adisplay screen. Meanwhile, displaying such a lesion as that appears onlyfor a short period of time and soon disappears from the screen onpurpose enables notification of presence of the lesion that is easilyoverlooked to the user.

Hence, the plurality of classifier candidates in accordance with thepresent embodiment may include a plurality of classifiers whose trainedmodels are identical and that have been subjected to differentpost-processing. In the case of the above-mentioned example, the firstclassifier candidate inputs time-series input images to the trainedmodel to acquire time-series outputs, and performs post-processing fordetermining whether or not a given lesion is consecutively detected inn1 frames based on the time-series outputs. The second classifiercandidate inputs time-series input images to the trained model toacquire time-series outputs, and performs post-processing fordetermining whether or not the given lesion is consecutively detected inn2 frames based on the time-series outputs. The n1 and the n2 areintegers that satisfy 1<n1<n2. This enables generation of the pluralityof classifier candidates based on one trained model.

As the pre-processing, a process of extracting a partial region of theinput image may be performed. Extracting the partial region can reducepixels serving as a processing target, and can thereby reduce processingtime. In a case where the processing time is maintained, processing timeper unit data amount is increased, and thus accuracy can be expected toincrease. Also in this case, there is a trade-off relationship betweenaccuracy and time similarly to the above-mentioned example, and which ofsettings is better depends on the user's preference. Hence,differentiating whether or not pre-processing for trimming the inputimage or differentiating whether or not a size or position of a regionis to be trimmed enables generation of the plurality of classifiercandidates based on one trained model.

Additionally, the pre-processing in the classifier candidate is notlimited to the above, and can include various kinds of processingperformed on the input image before being input to the trained model.Similarly, the post-processing for the classifier candidate is notlimited to the above, and can include various kinds of processingperformed on the output from the trained model.

Note that the above description has been given assuming that theprocessing such as the frame integration is the pre-processing or thepost-processing that is different from the processing executed in thetrained model. However, the trained model that includes the processingsuch as the frame integration may be generated. For example, the trainedmodel may be a model that receives images in a plurality of frames asinput data, performs a noise reduction process corresponding to theframe integration, and then outputs a detection result regarding theregion of interest. That is, the pre-processing or the post-processingmentioned herein may be implemented as processing that is different fromthe processing executed in the trained model, or may be implemented asprocessing that is executed in the trained model.

As described above, the plurality of classifier candidates is mutuallydifferent in processing method with respect to the input image. Theprocessing method mentioned herein corresponds to the pre-processingserving as processing in a former stage of the trained model, thepost-processing serving as processing in a subsequent stage of thetrained model, or both of the pre-processing and the post-processing.Accordingly, differentiating the processing with respect to the inputimage enables implementation of the plurality of classifier candidatesthat outputs different detection results.

For example, as described above, the plurality of classifier candidatesmay include two or more classifier candidates whose trained models areidentical and that have been subjected to different pre-processing orpost-processing. The trained model being identical means both the modeland the learning data being identical. However, the trained model is notlimited to the identical trained model, and the plurality of classifiercandidates may include two or more classifier candidates whose trainedmodels are different and that have been subjected to differentpre-processing or post-processing. The trained models being differentmeans at least one of the model or the learning data being different.

2. 5 Difference in Performance of a Plurality of Classifier Candidates

As described above, the plurality of classifier candidates in accordancewith the present embodiment is different in at least one of the learningdata, the model, the pre-processing, or the post-processing. Thisdifference enables differentiation of performance of the classifiercandidates.

The plurality of classifier candidates in accordance with the presentembodiment is, for example, mutually different in detection sensitivityto the presence/absence of the region of interest. The detectionsensitivity to the presence/absence represents easiness to detect theregion of interest. The sensitivity being high means that the region ofinterest included in the input image is easily detected, and is lesslikely to be overlooked. However, in a case where the sensitivity ishigh, there is a higher probability for false detection of a region thatis not the region of interest as the region of interest. In a case wherethe sensitivity is low, the false detection of the region of interestcan be prevented, but there is a higher probability for overlooking ofthe region of interest. Note that a recall rate, which will be describedlater, can be utilized as an index representing the detectionsensitivity.

This allows outputting of the detection result from the classifierhaving a detection sensitivity that is suited to user's preference. Forexample, it is possible to switch display depending on whether the useris a user who thinks that prevention of overlooking is important or auser who thinks that prevention of false detection is important. In acase where the region of interest is detected by the classifier, forexample, an object that improves viewability of the region of interestis displayed similarly to FIGS. 9 and 10 , which will be describedlater. In this case, a classifier candidate having a lower sensitivitymay possibly be selected for the reason that the user does not likedisplay of multitudes of objects on the display screen.

For example, in a case where the learning data for the first classifiercandidate includes multitudes of images regarding the given lesion andthe learning data for the second classifier candidate includes not somany images regarding the given lesion, it is assumed that the firstclassifier candidate has a higher detection sensitivity to the lesionand the second classifier candidate has a lower detection sensitivity tothe lesion. In this manner, there occurs a difference in detectionsensitivity to the presence/absence of the region of interest due to,for example, a difference in learning data.

Or else, since a processing algorithm in the forward direction isdifferent depending on a model, there exist a model that is suited todetection of a specific structure and a model that is not suited todetection of the structure. Hence, it is also conceivable that thereoccurs a difference in detection sensitivity to the presence/absence ofthe region of interest due to a difference in model.

Under a condition that a predetermined number of frames areconsecutively detected, a sensitivity to a lesion that is framed outsoon decreases. Execution of the frame integration increases asensitivity to a flat lesion. By trimming part of the input image, thedetection sensitivity can be expected to increase by the increasedaccuracy, but a sensitivity to a lesion in the periphery of the imagedecreases. In this manner, it is also conceivable that there occurs adifference in detection sensitivity to the presence/absence of theregion of interest due to a difference in pre-processing orpost-processing.

More specifically, the plurality of classifier candidates is mutuallydifferent in at least one of a detection sensitivity to a shape of theregion of interest, a detection sensitivity to a size of the region ofinterest, or a detection sensitivity to a color of the region ofinterest.

The detection sensitivity to the shape being different means that, forexample, when consideration is given to the region of interest having agiven shape, a detection sensitivity of the first classifier candidateis higher and a detection sensitivity of the second classifier candidateis lower. Alternatively, in a case where first to n-th shapes, which aren types of shapes (n is an integer of 2 or more), are assumed as shapesof the region of interest, the detection sensitivity to the shape beingdifferent means that the first classifier candidate has a relativelyhigher detection sensitivity to a first shape than the other shapes, andthe second classifier candidate has a relatively higher detectionsensitivity to a second shape than the other shapes. Stillalternatively, when consideration is given to respective detectionsensitivities with respect to the first to n-th shapes, tendencies ofthe detection sensitivities are different between the first classifiercandidate and the second classifier candidate. The same applies to thesize and the color.

The difference in detection sensitivity to the shape, the size, and thecolor may be attributed to the learning data, the model, or thepre-processing and the post-processing, similarly to the above-mentionedexample. For example, the detection sensitivity to the shape (the size,or the color) can be changed depending on how much amount of imagesincluding the region of interest having a specific shape (size, orcolor) is included in the learning data. In addition, since there exista model that is suited to detection of the specific shape (size, orcolor), and a model that is not suited to detection for the specificshape (size, or color), changing the model can change the detectionsensitivity to the shape or the like. Execution of a process ofadjusting the shape, the size, or the color such as edge enhancement,affine transformation, and color conversion as the pre-processing or thepost-processing also can change the detection sensitivity to the shapeor the like.

The region of interest may be a region corresponding to a lesion. Theplurality of classifier candidates is mutually different in detectionsensitivity to medical classification of the lesion. The medicalclassification represents classification of the lesion from a medicalperspective, and may be classification of a lesion type itself, orclassification of a degree of malignancy of a specific lesion (diseasestage).

For example, as macroscopic classification of gastric cancer, known is amethod of classifying the lesion into any of type0 to type5. Inaddition, regarding type0 superficial cancer, classification such astype0-I as a protruding type, type0-II as a superficial type, type0-IIIas a depressed type is also known. Besides the above classification,various kinds of classification as medical classification are known, anda wide range of these kinds of classification is applicable to thepresent embodiment.

The detection sensitivity to the medical classification being differentmeans that, for example, when consideration is given to a given type ofthe lesion or a given degree of malignancy of the lesion, a detectionsensitivity of the first classifier candidate is higher and a detectionsensitivity of the second classifier candidate is lower. For example,the first classifier candidate has a higher detection sensitivity to thetype0-I, and the second classifier candidate has a lower detectionsensitivity to the type0-I. Alternatively, the detection sensitivity tothe medical classification being different means that the firstclassifier candidate has a relatively higher detection sensitivity to alesion of a first type or a first degree of malignancy, and the secondclassifier candidate has a relatively higher detection sensitivity to alesion of a second type or a second degree of malignancy. For example,the first classifier candidate has a higher detection sensitivity to thetype0-I than with respect to lesions of other degrees of malignancy, andthe second classifier candidate has a higher detection sensitivity tothe type0-III than to lesions of other degrees of malignancy.Alternatively, when consideration is given to a relationship among adetection sensitivity to the type0-I, a detection sensitivity to thetype0-II, and a detection sensitivity to the type0-III, the relationshipis different between the first classifier candidate and the secondclassifier candidate.

Preparing classifier candidates that are different in performancedepending on medical classification allows the user's preference to bemore reflected on the detection result. Note that the difference indetection sensitivity to the medical classification may be attributed tothe learning data, the model, or the pre-processing and thepost-processing, similarly to the above-mentioned example. For example,with use of multitudes of images including the lesion of the givendegree of malignancy for machine learning, it is possible to increase adetection sensitivity to the lesion of the given degree of malignancy.

In addition, the plurality of classifier candidates is mutuallydifferent in detection result depending on an imaging state when theinput image is captured. The imaging state mentioned herein represents,for example, brightness in imaging and a relative positionalrelationship between the insertion section 310 and the object at thetime of the imaging. The positional relationship between the insertionsection 310 and the object may be, for example, a distance between theleading end of the insertion section 310 and the object. Alternatively,the relative positional relationship may be an angle between the leadingend of the insertion section 310 and the object. The angle is, forexample, an angle formed between a surface representing the object andan axis representing a longitudinal direction of the insertion section310, and is an angle representing a state whether or not the insertionsection 310 is at a correct position with respect to the object. Theimaging state may include information that identifies a wavelength bandof a light source 352 used for imaging and information that identifieswhether a pigment has been sprayed.

A change in the imaging state changes characteristics such as theposition, size, and brightness of the region of interest on the image.Hence, it is not easy to create a classifier candidate that can be usedfor various kinds of imaging states for general purpose. Preparing theclassifier candidates that are different in performance depending on theimaging state allows the user's preference to be more reflected on thedetection result. Note that the difference in detection sensitivity tothe imaging state may be attributed to the learning data, the model, orthe pre-processing and the post-processing, similarly to theabove-mentioned example. For example, with use of multitudes of imagescaptured in a given imaging state for machine learning, it is possibleto increase a detection sensitivity in the imaging state.

3. Display example of performance information Subsequently, a specificexample of performance information displayed by the performanceinformation processing section 120 is described. The performanceinformation in accordance with the present embodiment may be, forexample, comparison information indicating a difference in performanceof the plurality of classifier candidates. With use of the comparisoninformation, the user can easily recognize a difference in performanceof two or more classification candidates from the display screen, andcan thereby easily select a classifier candidate that is suited tohis/her preference. The following description is also given of theexample in which the performance information is the comparisoninformation, but the performance information in accordance with thepresent embodiment is not prevented from being information indicatingperformance of a given classifier candidate alone. As the informationindicating performance, various kinds of information, such as adetection result based on the input image, which will be describedlater, a detection result based on test data, and the learning data, canbe used. The following description is given of an example of displayingperformance information of two classifier candidates, but three or moreclassifier candidates may serve as a display target.

3. 1 Display of Detection Result Based on Input Image

FIG. 9 illustrates an example of a screen displaying the comparisoninformation. As illustrated in FIG. 9 , the performance informationprocessing section 120 performs a process of displaying respectivedetection results from the plurality of classifier candidates as theperformance information.

For example, the classification section 110 acquires an input image fromthe image acquisition section 331. The classification section 110 readsout the first classifier candidate from the storage section 333 andinputs the input image to the first classifier candidate to acquire afirst detection result. The first detection result is, for example,information that identifies the position or size of the region ofinterest on the input image. The first detection result may beinformation that identifies a rectangular region including the region ofinterest in a limited sense. In addition, the classification section 110reads out the second classifier candidate from the storage section 333and inputs the input image to the second classifier candidate to acquirea second detection result. The second detection result is, for example,information that identifies a rectangular region including the region ofinterest.

The classification section 110 outputs the first detection result, thesecond detection result, and the input image to the performanceinformation processing section 120. The performance informationprocessing section 120 performs a process of causing the display section340 to display an image in which the first detection result and thesecond detection result are superimposed on one input image. Note thatthe process of performing display by superimposing the detection resultson the input image overlaps with processing performed by the displaysection 340 after selection of the classifier by the user. Thus, in acase where an image illustrated in FIG. 9 is displayed, the displayprocessing section 336 may also serve as the performance informationprocessing section 120.

In FIG. 9 , the first detection result from the first classifiercandidate corresponds to A1. The second detection result from the secondclassifier candidate corresponds to A2 and A3. A1 and A2 are informationindicating the identical region of interest. That is, it is obvious froma screen in FIG. 9 that the second classifier candidate has detected tworegions of interest, and the first classifier candidate has detectedonly one region of interest out of the two regions of interest. Forexample, a user who thinks that a degree of importance of the region ofinterest corresponding to A3 is high or thinks that there is apossibility that she/he may overlook the region of interest withoutdisplay of the region of interest, it is assumed that he/she will selectthe second classifier candidate. In addition, a user who thinks that adegree of importance of the region of interest corresponding to A3 islow or thinks that there is a possibility that that she/he will notoverlook the region of interest without display of the region ofinterest, it is assumed that he/she will select the first classifiercandidate.

FIG. 10 illustrates another example of a screen displaying respectivedetection results as the comparison information. As illustrated in FIG.10 , the performance information processing section 120 may display afirst image in which the first detection result from the firstclassifier candidate is superimposed on the input image and a secondimage in which the second detection result from the second classifiercandidate is superimposed on the input image side by side. In otherwords, display screens illustrated in FIG. 10 corresponds to displayscreens obtained by dividing the display screen illustrated in FIG. 9into two. Also in this case, it is obvious that the second classifiercandidate has detected two regions of interest, and the first classifiercandidate has detected only one region of interest out of the tworegions of interest. This allows the user to grasp performance of eachclassifier candidate and select a classifier candidate that is suited tothe user's preference.

As illustrated in FIGS. 9 and 10 , the performance informationprocessing section 120 performs a process of simultaneously displayingrespective detection results with respect to the identical input imagefrom the plurality of classifier candidates as the performanceinformation. This enables simultaneous presentation of detection resultsfrom the plurality of classifier candidates using one screen, andthereby allows the user to easily make comparison and selection.However, the performance information processing section 120 may displaythe respective detection results from the plurality of classifiercandidates at different timings. For example, the performanceinformation processing section 120 may display the first image and thesecond image illustrated in FIG. 10 in a time-division manner.

In a case where the screen illustrated in FIG. 9 or FIG. 10 isdisplayed, a specific input image is required. Hence, for example, aprocessing procedure in this case is different from that illustrated inFIG. 3 . First, the observation mode is started, the input image isacquired in the observation mode, and thereafter the performanceinformation is displayed and the user's selection is received.Alternatively, an input image captured in an observation mode that hasbeen previously performed is stored in the storage section 333, and thescreen illustrated in FIG. 9 or FIG. 10 may be displayed using the inputimage. In this case, it is possible to display the performanceinformation before the start of the observation mode as illustrated inFIG. 3 .

3.2 Display of List of Detectable Lesions.

The performance information may be information that is acquired usingtest data. The test data is data that includes a plurality of testimages, and in which information that identifies a lesion in each testimage is known. The information that identifies the lesion is thepresence/absence of the lesion, the position of the lesion, a result ofclassifying the lesion, and the like. That is, inputting the test imageto the classifier candidate enables determination of whether or not thelesion included in the test image has been successfully andappropriately detected, or determination of whether or not a region thatis not the lesion has been falsely detected as the lesion. Note that thetest data may be evaluation data used at a learning stage, or may bedata that is different from the evaluation data.

FIG. 11 illustrates an example of a screen displaying the performanceinformation based on the test data. In this example, consideration isgiven to, for example, first to sixth lesions indicated in B1 to B6 aslesions included in the test data. Respective test images including thefirst to sixth lesions are input to each of the plurality of classifiercandidates. Note that the process of inputting the test images to eachclassifier candidate and acquiring the detection result may be executedin the classification section 110, or may be executed in an externaldevice outside the diagnosis support system 100.

For example, the first classifier candidate mentioned herein hassuccessfully detected the first to fourth lesions and the sixth lesion,but has failed to detect the fifth lesion. In addition, the secondclassifier candidate has successfully detected the third to sixthlesions, but has failed to detect the first and second lesions.

The performance information processing section 120 performs a process ofdisplaying a list of a plurality of regions of interest that isdetectable by each of the plurality of classifier candidates. In theabove-mentioned example, five lesions of the first to fourth lesions andthe sixth lesion are displayed as a list with respect to the firstclassifier candidate. Similarly, four lesions of the third to sixthlesions are displayed as a list with respect to the second classifiercandidate. In the example illustrated in FIG. 11 , the lesions that havebeen successfully detected by each classifier candidate are displayed ina relatively bright manner. This enables presentation of to what type oflesion detection each classifier candidate is suited using specificlesions, and can thereby facilitate the user's selection.

For example, the region of interest corresponds to the lesion, and theperformance information processing section 120 may perform a process ofdisplaying a list of a plurality of lesions that is detectable by eachof the plurality of classifier candidates, regarding each of aprotruding lesion, a flat lesion, and a depressed lesion. In the exampleillustrated in FIG. 11 , the first to third lesions indicated in B1 toB3, respectively, each correspond to the protruding lesion, the fourthand fifth lesions indicated in B4 and B5, respectively, each correspondto the flat lesion, and the sixth lesion indicated in B6 corresponds tothe depressed lesion. This enables display of a list indicatingdifferences in performance in accordance with a shape and type of thelesion, and can thereby facilitate the user's selection. In the examplein FIG. 11 , it is found that the first classifier candidate is suitedto detection of the protruding lesion, and the first classifiercandidate is suited to detection of the flat lesion and the depressedlesion.

As illustrated in FIG. 11 , the performance information processingsection 120 may display a list indicating whether or not each classifiercandidate has successfully detected each of a plurality of lesionsincluded in the test images. For example, the display also includesdisplay of the first classifier candidate having failed to detect thefifth lesion and the second classifier candidate having failed to detectthe first and second lesions. In the example illustrated in FIG. 11 ,the lesions that have failed to be detected by each classifier candidateare displayed in a relatively dark manner. This enables presentation ofto what kind of lesion detection each classifier candidate is not suitedusing specific lesions, and can thereby facilitate the user's selection.For example, with respect to the protruding lesion, the display clearlyindicates that the first classifier candidate has successfully detecteda plurality of protruding lesions, while the second classifier candidatehas failed to detect some of the plurality of protruding lesions, andcan thereby facilitate the user's selection.

3.3 Display of Data Indicating Classification Performance

The description has been given of the example of displaying the list ofthe information regarding whether or not specific lesions have beensuccessfully detected by each classifier candidate, with reference toFIG. 11 . However, the method for displaying the performance informationbased on the test data is not limited thereto.

FIG. 12A illustrates an example of a screen displaying data indicatingclassification performance as the performance information. As describedabove, classification results such as the shape, degree of malignancy,and size of the lesion included in the test image may be associated withthe test image. For example, assume that data including x1 protrudinglesions, x2 flat lesions, and x3 depressed lesions has been acquired astest data. By inputting the test data to the first classifier candidate,obtained is a detection result indicating that y1 protruding lesions, y2flat lesions, and y3 depressed lesions have been appropriately detected.By inputting the identical test data to the second classifier candidate,obtained is a detection result indicating that z1 protruding lesions, z2flat lesions, and z3 depressed lesions have been appropriately detected.

The performance information processing section 120, for example,acquires a recall rate as data indicating the classificationperformance. The recall rate represents, assuming that the number ofcases in which the lesion has been successfully and correctly detectedas the lesion is A and the number of cases in which the lesion has beenfalsely detected as being not the lesion is B, a ratio of A to A+B. Therecall rate is an index that enables identification of a ratio ofoverlooking. In the above-mentioned example, a recall rate of the firstclassifier candidate with respect to the protruding lesion correspondsto y1/x1, a recall rate of the first classifier candidate with respectto the flat lesion is y2/x2, and a recall rate of the first classifiercandidate with respect to the depressed lesion corresponds to y3/x3. Arecall rate of the second classifier candidate with respect to theprotruding lesion corresponds to z1/x1, a recall rate of the secondclassifier candidate with respect to the flat lesion is z2/x2, and arecall rate of the second classifier candidate with respect to thedepressed lesion corresponds to z3/x3.

In the example in FIG. 12A, the recall rate of the first classifiercandidate and the recall rate of the second classifier candidate aredisplayed for each lesion type. This enables display of whether eachclassifier candidate is suited to detection of the protruding lesion,the flat lesion, and the depressed lesion using a statistical indexvalue in an easily understood manner.

The performance information processing section 120 may display therecall rate of each classifier candidate with respect to each medicalclassification of the lesion, as illustrated in FIG. 12B. Alternatively,the performance information processing section 120 may display therecall rate of each classifier candidate with respect to each size ofthe lesion, as illustrated in FIG. 12C. This enables display of to whatkind of lesion detection each classifier candidate is suited in aneasily understood manner. The performance information processing section120 may display the recall rate in accordance with another conditionsuch as a color of the lesion.

Display of data indicating the classification performance is not limitedto display using a graph. FIG. 13 illustrates another example of ascreen displaying data indicating the classification performance. Asillustrated in FIG. 13 , the performance information processing section120 may use a table to display the recall rate of each classifiercandidate with respect to each type of the lesion. Also in this case, itis possible to display to what kind of lesion detection each classifiercandidate is suited using a statistical index value in an easilyunderstood manner.

The above description has been given of the example using the recallrate as the data indicating the classification performance. However, theperformance information processing section 120 may display another indexsuch as specificity and a rate of correct answers. The specificityrepresents, assuming that the number of cases in which a region that isnot the lesion has been correctly detected as being not the lesion is Cand the number of cases in which the region that is not the lesion hasbeen falsely detected as the lesion is D, a ratio of C to C+D. Thespecificity can be used as an index for determining whether or not theregion that is not the lesion is suspected indiscriminately.

The rate of correct answers represents a ratio of A+C to a total numberof cases. As described above, A is the number of cases in which thelesion has been successfully and correctly detected as the lesion, and Cis the number of cases in which the region that is not the lesion hasbeen correctly detected as being not the lesion. The total number ofcases corresponds to A+B+C+D. The rate of correct answers represents aratio of determination that has been made correctly, and thus serves asa simple and easy-to-understand index.

As described above, the performance information processing section 120may perform a process of displaying the classification performance ofthe plurality of classifier candidates as the performance information.The data indicating the classification performance is, for example, astatistical index, and may be the recall rate, the specificity, or therate of correct answers as described above. This enables presentation ofthe classification performance of the classifier candidate using theeasy-to-understand index to the user.

At this time, the performance information processing section 120 mayperform a process of displaying data indicating respective detectionresults from the plurality of classifier candidates with respect to thetest data as the data indicating the classification performance of theplurality of classifier candidates. The test data is data fordetermining the classification performance, and the informationregarding the presence/absence of the region of interest, the positionof the region of interest, the size of the region of interest, and theclassification result of the region of interest has been known, asdescribed above. With use of the test data, it is possible toappropriately determine the classification performance of the classifiercandidate.

3.4 Display Based on Previous Examination Result

The input image is an in-vivo image in which the living body iscaptured, and the performance information processing section 120 maydisplay data indicating appropriateness of a detection result inprevious examination using an in-vivo image as the data indicating theclassification performance of the plurality of classifier candidates.

For example, after a given classifier candidate is selected by theuser's selection as the classifier, the detection result using theclassifier is displayed. In a case where the user performs some kind ofaction on the display, the action is fed back, whereby the dataindicating the classification performance is generated.

The feedback made by the user is, for example, an input regardingresection of the living body and a result of pathologic analysis on theresected living body. For example, the user resects and removes a regionpresented by the classifier as the lesion using a treatment tool such asan energy device. The pathologic analysis is performed on the resectedportion of the living body. The user feeds back information indicatingwhether or not the resected portion of the living body is the lesion aspresented by the classifier based on the result of the pathologicanalysis. For example, the performance information processing section120 obtains a ratio of the number of resected portions of living bodiesbeing the lesion as indicated by the detection result with respect to atotal number of resections, and can thereby obtain an index similar tothe above-mentioned rate of correct answers.

Alternatively, the feedback from the user may be information indicatinga relationship between a region that is designated by the user as atarget of treatment such as resection and a lesional region presented bythe classifier. More specifically, when performing a treatment on aregion that has not been presented as the lesion by the classifier, theuser feeds back information indicating to this effect. This correspondsto the fact that the lesion that had failed to be detected by theclassifier has been uniquely detected and treated by the user. Thus, theperformance information processing section 120 is capable of obtaining,based on the feedback from the user, an index representing a ratio ofdetermination that has been falsely made as the lesion being not thelesion.

Note that the data indicating appropriateness of the detection result inthe previous examination can be obtained as numeric value data similarlyto the example using the test data. Hence, the performance informationprocessing section 120 may perform a process of using a graph asillustrated in each of FIGS. 12A to 12C or a process of using a table asillustrated in FIG. 13 to display the data indicating appropriateness ofthe detection result in the previous examination.

3. 5 Display of Learning Data

The above description has been given of the example in which theperformance information is a detection result itself obtained byinputting of some kind of image to the classifier candidate, orinformation obtained based on the detection result. However, theperformance information in accordance with the present embodiment is notlimited to information that requires acquisition of the detectionresult.

FIG. 14 illustrates another example of a screen displaying theperformance information. As illustrated in FIG. 14 , the performanceinformation processing section 120 may perform a process of displayinglearning data used for learning of the plurality of classifiercandidates as data indicating the classification performance of theplurality of classifier candidates. In FIG. 14 , a ratio of each of theprotruding lesion, the flat lesion, and the depressed lesion included inthe learning data with respect to each of the first classifier candidateand the second classifier candidate is displayed with a circle graph. Inthe example illustrated in FIG. 14 , it can be found that the learningdata of the first classifier candidate has a higher ratio of theprotruding lesion, and the learning data of the second classifiercandidate has a higher ratio of the flat lesion. Thus, the user canpresume that the first classifier candidate is suited to detection ofthe protruding lesion, and the second classifier candidate is suited todetection of the flat lesion.

As described above, the display mode for the performance information inaccordance with the present embodiment can be modified in variousmanners. For example, using the specific detection results asillustrated in FIGS. 9 to 13 is advantageous in that accuracy of theperformance information of the classifier candidate can be increased. Incontrast, using the learning data as illustrated in FIG. 14 isadvantageous in easiness of generation and display of the performanceinformation because a specific detection result is unnecessary.

4. Modification

Some modifications will be described below.

The user's selection receiving section 130 may be capable of receivingthe user's selection for selecting two or more classifier candidatesamong the plurality of classifier candidates. The classification section110 performs a process of selecting the classifier from the two or moreclassifier candidates selected by the user. This allows the diagnosissupport system 100 to support selection of the classifier in a casewhere the user cannot narrow down to one classifier candidate by merelyseeing the performance information.

The classification section 110 may select the classifier based on adegree of reliability representing a degree of probability of thedetection result. For example, in a case where a learning modelclassifies two regions including a lesional region and a normal region,the output layer of the trained model outputs a degree of probabilitythat a target region is the lesional region and a degree of probabilitythat the target region is the normal region. In a case where the outputlayer is a publicly-known softmax layer, the degree of probability thatthe target region is the lesional region and the degree of probabilitythat the target region is the normal region are such probability data asthat a sum of them becomes 1. For example, in a case where theprobability that the target region is the lesional region is a giventhreshold or more, the classification section 110 determines that thelesion has been detected, and outputs the detection result.

Assume that the first classifier candidate and the second classifiercandidate are selected from three or more classifier candidates based onthe user's selection. The classification section 110, for example,performs a process of inputting the input image to each of the firstclassifier candidate and the second classifier candidate in theobservation mode. The classification section 110 compares a degree ofreliability representing the degree of probability of the lesionalregion output from the first classifier candidate and a degree ofreliability output from the second classifier candidate with each other,and performs a process of automatically selecting a classifier candidatehaving a larger value as the classifier. This enables automaticselection of the classifier candidate having a higher degree ofreliability.

The user's selection receiving section 130 may receive selection ofclassifiers made by a plurality of users. For example, the diagnosissupport system 100 in accordance with the present embodiment isinstalled in each hospital, and a plurality of classifier candidates isdifferent for each hospital. The plurality of classifier candidatesbeing different means that not all the classifier candidates are matchedwith each other, and part of the classifier candidates may overlap witheach other as the above-mentioned original classifier. Since classifiercandidates in a given hospital include, for example, the customizedclassifier candidate, the diagnosis support system 100 is considered tohave characteristics that are suited to the hospital. However, it isassumed that a plurality of users having different preferences utilizesthe diagnosis support system 100 even within one hospital. In thisrespect, allowing the plurality of users to individually selectclassifiers enables switching of a detection result that is suited tothe user's selection even within a group using one diagnosis supportsystem 100. Specifically, the classification section 110 outputs thedetection result from the classifier in accordance with the user.

At this time, the storage section (for example, the storage section 333)may store a selection history of classifiers for each user. This enableschange of a default classifier for each user, and can thereby reduce aburden on a user regarding selection of a classifier for the second andsubsequent times.

Before observation of the input image by the user, the user's selectionreceiving section 130 may receive, after receiving the user's firstselection of the classifier, the user's second selection during theuser's observation of the input image.

FIG. 15 is a flowchart for describing processing of the presentmodification. Steps S301 to S305 in FIG. 15 are similar to steps S101 toS105 in FIG. 3 , respectively. That is, the diagnosis support system 100can receive the first selection, which is the user's selection forselecting the classifier, before the start of the observation mode.

After the start of the observation mode in step S305, in step S306, thecontrol section 332 of the endoscope system 300 determines whether ornot an operation mode of the diagnosis support system 100 is aclassifier correction mode.

In a case of YES in step S306, in step S307, the performance informationprocessing section 120 performs a process of displaying the performanceinformation of classifier candidates. In step S308, the user's selectionreceiving section 130 performs a process of receiving the user'sselection for selecting any of the classifier candidates. Theclassification section 110 identifies the classifier candidate selectedby the user as the classifier used for outputting of the detectionresult.

In a case of NO in step S306, the user's selection receiving section 130does not receive the second selection, which is the user's selection forcorrecting the classifier. In this case, for example, the classificationsection 110 identifies the classifier candidate selected in the firstselection or the default classifier candidate as the classifier used foroutputting of the detection result.

After the processing in step S308 or in a case of NO in step S306, instep S309, the diagnosis support system 100 resumes the operation in theobservation mode.

This enables reception of the user's selection multiple times at thetime of single observation for detecting the lesion. Receiving the firstselection first before the observation enables smooth start of displayof the detection result when the lesion appears in the image. Inaddition, receiving the second selection during the observation enables,in a case where an actual detection result is not matched with theuser's preference, switching of the detection result in accordance withthe preference. Note that performance information displayed for thefirst selection and performance information displayed for the secondselection may be identical information, or may be different information.Note that correction of the classifier may be executed twice or moreduring the single observation.

The plurality of classifier candidates in accordance with the presentembodiment may include two or more classifier candidates generated bymaking different settings to a common trained model. A setting made tothe trained model is, specifically, a threshold used at the time ofdetermination that the region of interest has been detected. Forexample, as described above, the trained model outputs the degree ofprobability that the target is the region of interest. In a case wherethe degree of probability is the threshold or more, the trained modeldetermines that the region of interest has been detected. Changing thethreshold can change a detection sensitivity to the region of interest.That is, changing the setting of the trained model can facilitategeneration of the plurality of classifier candidates having differentdetection sensitivities. At this time, since the performance informationis displayed in the method in accordance with the present embodiment, itis possible to present a difference in detection result with a change inthreshold to the user in an easily understood manner.

Although the embodiments to which the present disclosure is applied andthe modifications thereof have been described in detail above, thepresent disclosure is not limited to the embodiments and themodifications thereof, and various modifications and variations incomponents may be made in implementation without departing from thespirit and scope of the present disclosure. The plurality of elementsdisclosed in the embodiments and the modifications described above maybe combined as appropriate to implement the present disclosure invarious ways. For example, some of all the elements described in theembodiments and the modifications may be deleted. Furthermore, elementsin different embodiments and modifications may be combined asappropriate. Thus, various modifications and applications can be madewithout departing from the spirit and scope of the present disclosure.Any term cited with a different term having a broader meaning or thesame meaning at least once in the specification and the drawings can bereplaced by the different term in any place in the specification and thedrawings.

What is claimed is:
 1. A diagnosis support system comprising aprocessor, the processor being connected to a plurality of classifiersthat are different in performance; the processor displaying performanceinformation of each of the plurality of classifiers side by side;receiving a user's selection of the performance information displayedside by side; and inputting an input image to a corresponding one of theplurality of classifiers, the corresponding one being associated withthe performance information selected by the user.
 2. The diagnosissupport system as defined in claim 1, the performance information beingcomparison information that indicates a difference in performance amongthe plurality of the classifiers.
 3. The diagnosis support system asdefined in claim 1, the plurality of classifiers having mutuallydifferent detection sensitivities to absence or presence of a region ofinterest.
 4. The diagnosis support system as defined in claim 3, theplurality of classifiers having at least one of the mutually differentdetection sensitivities to a shape of the region of interest, themutually different detection sensitivities to a size of the region ofinterest, or the mutually different detection sensitivities to a colorof the region of interest.
 5. The diagnosis support system as defined inclaim 3, the region of interest being a region corresponding to alesion, and the plurality of classifiers having mutually differentdetection sensitivities to medical classification of the region ofinterest.
 6. The diagnosis support system as defined in claim 1, theplurality of classifiers outputting mutually different detection resultsdepending on an imaging state when the input image is captured.
 7. Thediagnosis support system as defined in claim 1, the plurality ofclassifiers using mutually different processing methods with respect tothe input image.
 8. The diagnosis support system as defined in claim 1,the plurality of classifiers including an original classifier and acustomized classifier created by the user.
 9. The diagnosis supportsystem as defined in claim 8, the customized classifier being createdbased on machine learning using learning images including an image heldby the user.
 10. The diagnosis support system as defined in claim 1, theprocessor performing a process of displaying respective detectionresults of the plurality of classifiers as the performance information.11. The diagnosis support system as defined in claim 10, the processorperforming a process of simultaneously displaying respective detectionresults of the plurality of classifiers with respect to an identicalinput image as the performance information.
 12. The diagnosis supportsystem as defined in claim 1, the processor performing a process ofdisplaying a list of a plurality of regions of interest that isdetectable by each of the plurality of classifiers.
 13. The diagnosissupport system as defined in claim 12, each region of interest being aregion corresponding to a lesion, and the processor performing a processof displaying a list of a plurality of the lesions that is detectable byeach of the plurality of classifiers, regarding each of a protrudinglesion, a flat lesion, and a depressed lesion.
 14. The diagnosis supportsystem as defined in claim 1, the processor performing a process ofdisplaying data indicating classification performance of the pluralityof classifiers as the performance information.
 15. The diagnosis supportsystem as defined in claim 14, the processor performing a process ofdisplaying data indicating respective detection results of the pluralityof classifiers with respect to test data as the data indicating theclassification performance of the plurality of classifiers.
 16. Thediagnosis support system as defined in claim 14, the processorperforming a process of displaying learning data used for learning ofthe plurality of classifiers as the data indicating the classificationperformance of the plurality of classifiers.
 17. The diagnosis supportsystem as defined in claim 14, the input image being an in-vivo image inwhich a living body is captured, and the processor performing a processof displaying data indicating appropriateness of a detection result inprevious examination using the in-vivo image as the data indicating theclassification performance of the plurality of classifier candidates.18. The diagnosis support system as defined in claim 1, the processorbeing capable of receiving the user's selection for selecting two ormore classifiers among the plurality of classifiers. and the processorperforming a process of selecting a classifier among the two or moreclassifiers selected by the user.
 19. The diagnosis support system asdefined in claim 1, the processor receiving selection of the respectiveclassifiers by a plurality of the users, and outputting a detectionresult from each of the classifiers in accordance with the user.
 20. Thediagnosis support system as defined in claim 1, the processor receivingthe user's first selection of a corresponding one of the classifiersbefore the user's observation of the input image, and thereafterreceiving the user's second selection of a corresponding one of theclassifiers during the user's observation of the input image.
 21. Thediagnosis support system as defined in claim 1, the plurality ofclassifier includes two or more of the classifiers generated by makingdifferent settings to a common trained model.
 22. The diagnosis supportsystem as defined in claim 1, the processor displaying at least twotypes of performance that are in a trade-off relationship as theperformance information.
 23. The diagnosis support system as defined inclaim 22, the performance in the trade-off relationship is detectionaccuracy and processing time, or detection accuracy and a frequency ofdisplaying a detection result.
 24. A diagnosis support methodcomprising: presenting performance information that is informationregarding performance of a plurality of classifiers, the plurality ofclassifiers outputting mutually different detection results whendetecting a region of interest from an input image; receiving a user'sselection for selecting at least one of the plurality of classifiers asa classifier serving as an output target; and outputting a detectionresult of the classifier selected by the user's selection, thepresenting including presenting at least two types of performance thatare in a trade-off relationship as the performance information.
 25. Astorage medium storing a diagnosis support program that causes acomputer to implement: presenting performance information that isinformation regarding performance of a plurality of classifiers, theplurality of classifiers outputting mutually different detection resultswhen detecting a region of interest from an input image; receiving auser's selection for selecting at least one of the plurality ofclassifiers as a classifier serving as an output target; and outputtinga detection result of the classifier selected by the user's selection.